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Abstract — Cognitive relaying has been introduced for oppor- 
tunistic spectrum access systems by which a secondary node 
forwards primary packets whenever the primary link faces 
an outage condition. For spectrum sharing systems, cognitive 
relaying is parametrized by an interference power constraint 
level imposed on the transmit power of the secondary user. For 
sensing-based spectrum sharing, the probability of detection is 
also involved in packet relaying control. This paper considers the 
choice of these two parameters so as to maximize the secondary 
nodes' throughput under certain constraints. The analysis leads 
to a Markov decision process using dynamic programming 
approach. The problem is solved using value iteration. Finally, 
the structural properties of the resulting optimal control are 
highlighted. 

Index Terms — packet relaying, spectrum sensing, spectrum 
sharing, dynamic programming, value iteration. 



I. Introduction 

Cognitive radio (CR) is a technology whereby the coex- 
istence of licensed users and unlicensed users on the same 
bandwidth can be achieved. Coexistence can be provided 
by implementing two different types of schemes known as 
opportunistic spectrum access (OSA) and spectrum sharing 
(SS). In OSA CR 1 1 1, a secondary user (SU) i.e., an unlicensed 
user, is allowed to access the spectrum, which originally 
allocated to the primary user (PU), only when the spectrum 
is not used by the PU. In an SS CR |2], the SU is allowed 
to transmit simultaneously with the PU in the same spectral 
band, as long as the interference from the SU does not degrade 
the performance of the PU to an unacceptable level. Clearly, 
both types of CR systems guarantee the main design principle 
of CR which can be expressed as follows: a secondary system 
should be "transparent" to the primary system, so as not to 
interfere with the licensed use of the spectrum. 

\ 

Research on CR technology includes several investigations 
of OSA and SS systems at both the physical (PHY) and 
medium access control (MAC) layers. Focusing on the PHY 
layer, an investigation of achievable capacity and optimal 
power allocation over fading channels in SS systems is found 
in ||3] while in |4| an analysis of capacity in OSA systems 
is provided. Moreover, in |J5l a new type of CR system 
called sensing-based SS has been devised and analyzed based 
on a capacity formulation. On the other hand, investigations 
dedicated to the MAC layer have focused on issues such 
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as the tradeoff between spectrum sensing and the achievable 
throughput per frame as studied in fSl, while in ff\, the joint 
design of the PHY and MAC layers based on the spectrum 
sensing results has been proposed. Finally, in |]8], stability 
issues of queues at the MAC layer have been investigated in 
OSA systems in which the relaying of primary packets can be 
supported by the secondary system. 

In this paper, we examine packet relaying in SS systems 
that (unlike the OSA system consider in |8|) provide re- 
laying capability of primary packets by the secondary sys- 
tem. This setting is motivated by the fact that the primary 
network can be served from the secondary network, albeit 
with competitive behavior due to the sharing nature of the 
SS system. Specifically, the system is a sensing-based SS; 
that is, an interference power constraint is imposed for the 
protection of PU in conjunction with spectrum sensing (SpSe). 
SpSe introduces more opportunities for simultaneous packet 
relaying and secondary transmission as pointed out in ||5] 
and m. Motivated by this factor, we model and analyze 
packet relaying in sensing-based SS systems, deriving the 
achievable primary and secondary throughputs, and showing 
their interdependencies on SpSe, power control (PoC), the 
interference power constraint and the utilization factors of 
the queues. Considering a time slotted system, the sensing 
results at time i — 1 will influence the achievable throughput 
at time t for both PU and SU assuming a specific interference 
power constraint level and primary utilization factor To this 
end, we specify a Markov decision process (MDP) in order 
to formulate the system's behavior and we proceed with the 
control of this behavior using a dynamic programming (DP) 
based solution. Henceforth, we define an immediate reward 
function that represents the secondary throughput and in the 
sequel we use value iteration in order to find a solution 
recursively. For relaxing the complexity of the problem, we 
investigate several scenarios individually. 

The rest of this paper is organized as follows. Section II 
describes the system model. Section III provides the system 
analysis, deriving the secondary and primary throughputs. 
In Section IV, we present the MDP model and Section V 
describes the value iteration process and the properties of the 
resulting control. Section VI provides a summary this work 
and some problems of interest for future work on this topic. 

II. System Model 

We assume a sensing-based SS system with one pair of 
SU-Tx and SU-Rx forming the secondary link and one pair of 



PU-Tx and PU-Rx forming the primary link with instantaneous 
channel power gains gs and gp respectively. These two links 
share the same frequency band and thus interference is as- 
sumed between them. The instantaneous channel power gains 
from the SU-Tx to the PU-Rx and from the PU-Tx to the SU- 
Rx are denoted by gsp and gps respectively. The considered 
system is depicted in Fig. 1 . Both links are assumed to be flat- 
fading channels with additive white Gaussian noises (AWGNs) 
denoted by no and ni at the PU-Rx and SU-Rx respectively, 
where uq and ni are assumed to be independent and with 
the distribution CA/'(0, A^o) (circularly symmetric complex 
Gaussian). PoC is employed at the SU-Tx in conjunction with 
spectrum sensing. PoC is deployed according to the rules 
of sensing-based SS systems whereby the transmit power is 
allocated with the water-filling algorithm when the channel 



is idle, denoted by P, 



(0) 



and with an additional interference 
[5] when the channel 



power constraint protecting the PU 
is busy denoted by PJ . The SpSe results in four different 
results known as false alarm and no false alarm under the 
hypothesis Hq when the channel state is idle with probability 
ttq; and detection and missed detection under the hypothesis 
Hi when the channel state is busy with probability tti. The 
corresponding conditional probabilities of the sensing results 
are denoted as Pf, (l—Pf), and Pd, {l~Pd) respectively. SpSe 
can be accomplished by an energy detector similar to the one 
presented in |6| with a sensing time r within a frame period 
T and a sensing threshold rj compared to an instantaneous 
sensed signal-to-noise-ratio (SNR) '^se- 

We also assume that the SU-Tx and the PU-Tx retain their 
own queues, denoted by Qs and Qp respectively, used for 
forwarding their own packets. In addition, the SU-Tx retains a 
queue for forwarding packets of the primary network, denoted 
by Qps- This is known as packet relaying capability by which, 
if the primary link is in outage condition, then the packets can 
be forwarded by the secondary node via Qps thereby helping 
emptying the queue of the primary user [SJ. Instead of the 
behavior of packet relaying in OSA systems as proposed in 
||8], in our case i.e. sensing-based SS, the primary packets 
in Qp can be forwarded to PU-Rx simultaneously with the 
secondary packets in Qs due to the properties of SS systems. 
Notably, we assume that when the primary packets in Qps are 
forwarded, the secondary packets in Qs are not transmitted. 

Thereafter, the behavior of our system is summarized as 
follows: 

- the PU-Tx forwards its own packets of Qp, through the 
primary link with gain gp as long as the link is not in outage. 

- the SU-Tx shares the primary channel used by PU-Tx and 
forwards its own packets of Qs, through the secondary link 
with gain gs as long as the link is not in outage. 

- the SU-Tx forwards primary packets of Qps, through the 
link with gain gsp as long as the primary link is in outage. 

We analyze below the system's behavior providing the 
corresponding formulation. 
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Fig. 1 . System model of a sensing-based SS system witli relaying capability. 

A. Secondary throughput 

The secondary throughput according to the spectrum sens- 
ing results and the transmit power is analysed as follows: 

- No false alarm and unconstrained transmission: If the 
SpSe function results in no false alarms when the PU is 
idle, then the transmission is accomplished unconstrained with 
transmit power Ps and cut-off level /3s with the following 
probability: 



fJ-sfiO = (1 - Pf)e 
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(1) 



- False alarm and constrained transmission: If the SpSe 
function results in false alarms when the PU is not active, then 
the transmission is accomplished constrained with transmit 
power Ps and cut-off level /3sp with the following proba- 
biUty: 



A*s,oi 






f 



ttq 



(2) 



- Missed detection and unconstrained transmission: If the 
SpSe function results in missed detection when the PU is ac- 
tive, then the transmission is accomplished unconstrained with 
transmit power Ps and cut-off level /3s with the following 
probability: 



Ms,io - (1 - Pd)e' -/(i+'-p^) 
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(3) 



- Detection and constrained transmission: If the SpSe 
function results in detection when the PU is active, then the 
transmission is accomplished constrained with transmit power 
Ps and cut-off level Psp defined as follows: 



III. System Analysis 

In this section, we provide the analysis of secondary and 
primary throughputs for the system model described above. 



UsSi^Pde' -'^/(^+-'ps)'ni 
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Finally, since the packet transmission will be done only in 
situations where there are available packets in the secondary 



queue, the frame utilization factor becomes (T — t)/T due to 
sending time, and the primary link is not in outage condition 
with probability e^'^p/'^'p, then the secondary throughput for 
each of the above cases is obtained as follows: 



_T-T Xs -bL 



(5) 



T ' fis 
where the xx index refers to the aforementioned scenarios. 

B. Primary throughput 

The primary throughput consists of two different parts, a 
fixed one related to the conditions of the primary link and its 
own queue and the other one is the relaying through secondary 
link which varies according to the sensing results. Thus, the 
overall primary throughput is defined as follows: 
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(6) 



^ i^ps .raax , 

where the first term represents the throughput of the PU 
achieved over its own link and thus encompasses the cases 
in which no outage is experienced on the primary link and 
the primary queue contains packets, and the second term is 
related to the current sensing result as long as the Q^s queue 
contains packets and the primary link experiences outage. As 
previously for the secondary throughput, the four possible 
primary relaying throughputs due to the different sensing 
results are defined below for each case: 

- Afe false alarm and unconstrained transmission: 
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- False alarm and constrained transmission: 

- Missed detection and unconstrained transmission: 



Mpsao = (1 - Pd)e 
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Detection and constrained transmission: 



(7) 



(8) 



(9) 



(10) 



C. Notes 

Notably, we assume in the analysis that the PU's activity tti 
is equal to the utilization factor \p/ jJLp^max of the Qp, indicat- 
ing that the channel is considered active as long as the primary 
queue Qp contains packets. Therefore, the primary user's idle 
probability is obtained as ttq = 1 — tti = 1 — Xp/fip.max- In 
this way, the primary and secondary throughputs are related 
through the utilization factor of Qp denoted as pp. The 
utilization factor pp is estimated by the SU-Tx by observing 
the fraction of idle slots, and measuring the ACK/NACK 
messages sent by the SU-Rx |8 1. Moreover, it is clear from the 
above that the utilization factor of the Qs denoted as ps and 
the transmit power at secondary link Pg will change the values 
of the achievable secondary throughput. Thus, we can model 



the system as a Markov decision process that will include 
these three parameters (i.e. pp, ps and Pg) as states and the 
probability of detection Pd and interference power constraint 
Ic as actions since they can be adapted by the SU-Tx. Details 
are given in the next section. 

IV. Markov Decision Process Definition 

In this section, we model the system dynamics as a Markov 
Decision Process giving first the state and action spaces and 
finally the dynamic programming problem formulation. 

A. State and action spaces 

The system consists of a finite set of states S G {pp, Ps,Ps) 
that express the utilization factor pp in the primary queue Qp, 
the utilization factor ps in the secondary queue Qs and the 
transmit power Pg at the SU-Tx. Moreover, the system is 
controlled through a finite set of actions A E {Pd,Ic) that 
includes the probability of detection Pd and the interference 
power constraint Ic- We are interested in the following condi- 
tions in each time slot for the system control: 



{pp,Ps,Ps;Pd,Ic) 



(11) 



that is, the current utilization factor pp in the primary queue 
Qp, the current utilization factor ps in the secondary queue Qs, 
the current transmit power P, at the SU-Tx and the probability 
of detection P^ and interference power constraint I^ used in 
the previous time slot. The controls applied (i.e. actions that 
should be taken) in each time slot are the pair of parameters 
{Pd,Ic) based on the calculation of the reward function. The 
reward function in our system is the secondary throughput ps- 
Therefore, the decisions made per slot are: what should be the 
probability of detection Pd and the interference power con- 
straint Ic level of the SpSe and PoC mechanisms respectively 
in order to maximize the ps- 

Based on this functionality, the considered system becomes 
a controlled Markov Decision Process 1 11 1 and in this case 
we can employ a dynamic programming recursion to derive 
and calculate the optimal control llT2l of packet relaying in 
sensing-based SS systems. 

B. Reward function and transition probability 

First, we define an immediate reward function 
g{pp, PsjPs'jPdjIc) which provides a measure of the 
maximum secondary throughput ps that can be achieved 
at the current time slot without consideration of the future 
throughput given the current utilization factor of the secondary 
queue ps, the current utilization factor of the primary queue 
Pp, the current current transmit power P, and the previous 
probability of detection P^ of SpSe and interference power 
constraint I^ of PoC. Afterwards, we define the reward 
function J{pp, ps, Pg', Pd, Ic)^ which provides a measure of 
the expected secondary throughput ps going from one state s 
to another state s with transition probability 



Pa{s,s)=Pr{. 
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where a G A is the action and s E S. This is the probability 
that action a in state s at time t will lead to state s' at time t+1. 
In the defined MDP, assuming the action of SpSe function and 
the activity of the PU (i.e. the utilization factor) and taking 
into account the sensing results |5|, the transition probability 
Pa(s,s ) is obtained as follows; 



Pais,s') 



ttqP/ , ifHo and SpSe gives Hi 
7ro(l — Pf) , if Ho and SpSe gives Hn 
7ri(l — Pd) , if Hi and SpSe gives Hq 
T^iPd, if Hi and SpSe gives Hi 



Notably, the probability Pa{s, s ) depends on the PU's activ- 
ity (i.e. the primary utilization factor pp) and thus implicitly on 
the state of the considered MDP and thus the decision making 
is accomplished recursively. 

C. Packet relaying control and costs 

The SU-Tx decides whether to forward the primary packets 
based on the control of the SpSe and PoC mechanisms 
changing the probability of detection and the interference 
power constraint respectively. However, this control comes 
with overhead costs. We assume the following interpretation 
as costs for our control approach: 

- Detection control cost: Increasing the detection probabil- 
ity results in a more costly detection due to the larger number 
of required samples among other factors. Accordingly a cost 
factor which is directly proportional to Pd is introduced. We 
define this cost as follows "^{Pd) = sPd where s is a constant. 

- Interference control cost: Reducing the interference power 
constraint of the PoC stresses the SU-Tx's transmission, hence 
a discount factor is introduced that is increased as Ic is 
decreased. The concept can be explained as follows: for more 
protection, less Ic is demanded, and the more the cost for SU- 
Tx transmission is due to a low power level [14|. We define 
this cost as ^(/c) = cPs where c is a constant. 

Below, we define the dynamic programming problem based 
on the aforementioned characteristics of the defined MDP. 

D. Dynamic programming problem formulation 

Having defined the MDP, the system becomes a controlled 
Markov chain and hence, we can develop a dynamic program- 
ming recursion to compute the optimal control as follows: 



J{pp, ps,Ps; Pd, Q = max{5(Pp, Ps,Ps; Pd, Ic) 

-^{Pd) ^ ^Ic) +Y.^Pais,s')J{pp,Ps,Ps;P'd,l'c)} (14) 



where the joint optimization {Pd,Ipk) is performed over the 
following conditions and selection sets, when the system is in 
state {pp,ps,Ps:,P'd,l'c)- 

1) the stabilities of all queues are retained i.e. Xp < p-p^nax, 

^ps ^ l^ps.max anu As ^ Ps.max, 

2) the transmit power is constrained over the range [0, Pav] 
through the PoC; 



3) the probability of detection is optimized over a continu- 
ous range Pd G [0, 1]; and 

4) the interference power is optimized over a discrete and 
bounded power range [Ic^min, Ic,max]- 

V. Value Iteration and Properties of Control 

Solving the DP recursion in (14) results in the optimal con- 
trol of the probability of detection and the interference power 
constraint when the system state is {pp,Ps, ps;Pd,Ic)- The 
solution can be obtained using the value iteration method lfT2l . 
We confirm the usage of value iteration with the following 
proposition. 

Proposition I: There exists a stationary optimal control 
solution of (14) obtainable by value iteration. 

Sketch of Proof: The DP terminates when the primary queue 
Qp empties, i.e., pp = 0. The proposed policy is based 
on the fact that the primary queue is emptied by the factor 
Pp introduced in secondary throughput via vri and thereby 
the primary queue Qp will be emptied in finite time. This 
condition guarantees the existence of a stationary optimal 
control 1121 . The full proof is omitted here for lack of space. 

Using standard value iteration, we start with an initial guess 
for the maximum reward function and plug it into the DP 
problem defined in (14). If we consider k iterations, the value 
iteration problem becomes as follows: 



j''+\pp, Ps,Ps; Pd, Ic) = max{5(pp, ps,Ps; Pd, /J 

-^{Pd) - $(/c) + Y. PPais, s')JHPp, Ps,Ps;P'd, I'c)} (15) 



where based on this formula the following value iteration 
algorithm is used: 

Algorithm 1 Value Iteration Algorithm 
Data: 

• Set Pa{s) equal to the transition probability matrix Vs G S. 

• For fc = 0, set J'^{s\ a) and j'''+^(s; a) equal to the reward 
matrix Vs E S and Va G A. 
while maxs | J'"'"'"^ - J'^ |< e do 

for each state s and Va do 
I calculate 7*^+^ based on ( fTSb 
end 

Set J'' = j'^+i; 
Set fc = fc + 1; 
end 

The iteration continues until convergence based on the 
adopted policy. Once we work out the solution, we store the 
secondary throughput as a lookup table. The size of the lookup 
table for each iteration k is 0{pp, p'^, Ps,Pd,Ic) where the 
size depends on the quantized levels of primary and secondary 
utilization factors, the quantized levels for the channel gain of 
the secondary link and thus the transmit power, the selected 
probability of detection and interference power constraint in 
each iteration. During the system operation, the SU-Tx can 
determine the optimal secondary throughput from the table 
according to real-time system parameters. 



To evaluate the performance of the proposed DP control ap- 
proach, we provide three suboptimal heuristic policies defined 
as follows lil4J. 



A. Variable probability of detection and fixed interference 
power constraint 

Let us consider the simple case in which the interference 
power constraint level is fixed and we need to optimize over 
the probability of detection. In this scenario, the probability of 
detection Pd is the only control parameter that can be varied 
to control the packet relaying process; then, we get the value 
iteration as follows: 



j'^^^iPp, Ps,Ps;Pd) = max{g(pp, ps,Ps;Pd) 

-*(p<i) + ^/?p,(s,s')J"(Pp,p.,Ps;Pd)} (16) 




B. Variable interference power constraint and fixed probabil- 
ity of detection 

We now consider the case in which the probability of 
detection is fixed Pd ~ 5 and thus the problem is defined 
as follows: 



J^^^iPp. Ps,Ps; Ic) = max{g(pp, ps, Ps'Jc) 
-Hh) + y^f3Pa{s,s')j''{pj,, Ps,Ps;h)} 



(17) 



where the decision is made only upon the interference power 
constraint Ic state space. 

It is interesting to explore the properties of the proposed 
control. In Fig. 2, we plot the secondary throughput ^j,s that 
represents the derived reward function J from the value 
iteration method versus the probability of detection Pd for a 
fixed average power at the secondary link equal to Pav = hdB, 
different levels of interference power constraints equal to 
Ic = —15dB, Ic = —5dB and Ic — 5dB, and utilization factor 
at the primary queue to be pp =0.1 and pp = 0.9. The sensing 
time is assumed equal to r = 0.3ms within a frame period of 
T — 1ms and the sensing threshold 77 is implicitly obtained for 
each Pd assuming a sensed SNR '^se = —l5dB. Notably, we 
do not follow any optimization over sensing time or threshold 
as has been done in |6| and |9|, and thus we choose arbitrary 
value for the sensing time since our optimization is over Pd 
and it is not related to the types of optimization presented in 
these two works. Moreover, the secondary utilization factor is 
obtained using A^ — 0.5 and p,s.max — 0.8. Upon examining 
the Fig. 2, it is clear that the reward function is maximized for 
all cases in values of Pd close to 0.7 and 0.8. It is also evident 
that as long as the interference power constraint is relaxed i.e. 
increased then the value iteration increases as well. Moreover, 
a high utilization factor pp limits the reward function; however, 
this effect is negligible in high values of interference power 
constraints. Notably, the sensing constant is s = 2 and the 
number of iterations requires were close to 83 for both pp 
values. 



Fig. 2. Secondary thi'oughput fis vs. probability of detection Pj, for average 
transmit power at the SU-Tx Pav = 5dB, for different levels of interference 
power constraints Ic = —15dB,—5dB,5dB and utilization factors pp = 
0,1.0.9. 



Fig. 3 depicts the results of optimization over the interfer- 
ence power constraint for a fixed probability of detection. We 
plot again the secondary throughput fig but vs. the interference 
power values for different probabilities of detection equal to 
Pd = 0.1 and Pd = 0.9 and different utilization factors with 
equal values as previously. Here, we can see that the control 
of the interference power constraint does not have any positive 
result for low probability of detection values, e.g. Pd = 0.1, 
and a positive one appears when high probabilities of detection 
are considered. This is explained from the fact that the benefit 
of interference power constraint is introduced in a spectrum 
sharing system when the detection probability is high. Notably, 
the interference control constant is c = 2 and the number of 
iterations required were close to 80 for both pp values. 

C. Joint control: variable probability of detection and variable 
interference power constraint 

In the case of joint control of the probability of detection and 
interference power constraint, the value iteration DP problem 
is formulated now as follows: 



J^'^'^iPp, Ps,Ps]P'd, I'c) = max{g(pp, p„ Ps]P'd) 

-^{Pd) - ^(/c) + J2 PP-^-'^ s')J\pp, Ps,Ps;P'd)} (18) 



This implementation differs from the previous one since 
now the cost-to-go function, i.e. the reward minus the costs, is 
controlled by the joint leverage of the probability of detection 
and interference constraint. 

Fig. 4 shows the values of control parameters, i.e. Ic and 
Pd, which maximize the secondary throughput for a specific 
average transmit power Pav at the secondary link when the 
joint control is accomplished. We see that for high power 
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Fig. 3. Secondaiy throughput /is vs. interference power constraint Ic, for 
average transmit power at the SU-Tx Pav = 5dB, for different probabilities 
of detection P^j = 0.1, 0.9 and utilization factors pp = 0.1, 0.9. 



We have analyzed the system highlighting the interdependen- 
cies between the system components, and we have defined a 
Markov decision process for modelling the system behavior, 
which was divided into a finite state space and a finite action 
space. The model resulted in a dynamic programming problem 
that was solved using the value iteration. The objective was 
the maximization of the secondary throughput by managing 
the spectrum sensing and the interference power constraint. 
Hence, we have designed and evaluated practical heuristics 
achieving optimal performance by first separating these two 
control parameters and then combining them. 

An interesting issue for further study based on the above 
analysis and results is policy improvement using policy it- 
eration finding exact policies under certain conditions and 
reinforcement learning for estimating the optimal action- state 
function without requiring a model of the environment. This 
approach should allow manipulation of the policies directly, 
rather than finding them indirectly via the optimal value 
function. Moreover, the cost for particular policies will play an 
additional role towards the depiction of optimal policies with 
the lowest costs. Finally, we will be able to use simulation to 
compare several optimal policies highlighting the performance 
capabilities of the proposed control in packet relaying. 
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Fig. 4. Values of control parameters Pa and Ic for a specific range of 
the average power Pay at the secondary link that maximize the secondary 
throughput. 



regions i.e. Pav > lOdB the probability of detection takes the 
value Prf = 0.7 for each interference power constraint value Ic- 
For low average regions, i.e. Pav < ~hdB, the maximization 
is achieved at P^ = for interference constrain equal to 
Ic < —lOdB and at P^; = 1 for interference constrain equal 
to Ic > —lOdB highlighting the important role of interference 
power constraint at the low power region. These conclusions 
can be confirmed by Fig.2 and Fig. 3 respectively in which the 
separate control has been presented. 

VI. Summary and Future Work 

In this paper, we have investigated the problem of packet 
relaying control in sensing-based spectrum sharing systems. 
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